Tarball all needed templates from different folder #214
Conversation
Pull Request Test Coverage Report for Build 10934406788Details
💛 - Coveralls |
There was a problem hiding this comment.
Really nice solutionwith the lockable set! Looks good to me in general but I haven't tried it (yet). One small concern are the non-flat structures in the tar ball but I might be wrong here.
One question not entirely related to this PR: Shouldn't the toyfiles be included here as well? Since we no allow toydata mode "read" they should be tarred as well or am I missing something?
| logger.warning( | ||
| "The template path contains subdirectories. All templates files will be tarred." | ||
| ) | ||
|
|
||
| def _tar_h5_files(self, directory, template_tarball="templates.tar.gz"): | ||
| """Tar all .h5 templates in the directory and its subdirectories into a tarball.""" |
There was a problem hiding this comment.
The docstring should be updated
There was a problem hiding this comment.
Yes, you are right.
| for dirpath, dirnames, filenames in os.walk(self.template_path): | ||
| for filename in filenames: | ||
| logger.info(f"File: {filename} in {dirpath}") | ||
| if self._contains_subdirectories(self.template_path): |
There was a problem hiding this comment.
_contains_subdirectories is no longer used after this PR sop we might as well delete it or am I missing something?
There was a problem hiding this comment.
Yes, you are right.
Toydata is added to the file in this line: alea/alea/submitters/htcondor.py Line 593 in ba824ac |
as long as their names are different
Asking all templates are in the same folder does not make sense, configuration like: https://github.com/XENONnT/nton/blob/fcccf25d583e1ee218724c4962d6fac9583c5191/nton/wimp/configs/sr0_sr1/wimp_statistical_model_ac_optimization.yaml#L409 and https://github.com/XENONnT/nton/blob/fcccf25d583e1ee218724c4962d6fac9583c5191/nton/wimp/configs/sr0_sr1/wimp_statistical_model_ac_optimization.yaml#L553 should be allowed.
LockableSetand initialize a variableTEMPLATE_RECORDSinalea/utils.pyTEMPLATE_RECORDSinalea.utils._prefix_file_path, soTEMPLATE_RECORDSwill record all needed templates absolute pathalea.submitters.htcondor.SubmitterHTCondor._validate_template_pathoralea.submitters.htcondor.SubmitterHTCondor._check_filename_uniqueany morealea.submitters.htcondor.SubmitterHTCondor._tar_h5_files, first copy all templates intotemplates_tarball_dirthen make tarball oftemplates_tarball_dir